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February  4,  2000 

The  Honorable  Dan  Miller 
Chairman 

The  Honorable  Carolyn  B.  Maloney 
Ranking  Minority  Member 
Subcommittee  on  the  Census 
Committee  on  Government  Reform 
House  of  Representatives 

The  U.S.  Census  Bureau  is  preparing  to  conduct  the  2000  decennial 
census-an  event  that  will  significantly  influence  the  lives  of  every  U.S. 
resident  and  constitutes  the  largest  peacetime  mobilization  event  in  our 
nation’s  history.  In  particular,  Census  will  be  staffing  1.35  million  temporary 
field  positions  to  capture  1.5  billion  pages  of  data  from  about  119  million 
households.  To  meet  this  important  and  massive  challenge,  the  bureau 
plans  to  rely  extensively  on  information  technology,  including  its  new  Data 
Capture  System  (DCS)  2000.  DCS  2000  is  to  be  operated  at  four  bureau  data 
capture  centers1  to  check  in,  digitally  image,  and  optically  read  the  data 
handwritten  onto  census  forms  and  convert  these  data  into  files  that  will  be 
transmitted  to  bureau  headquarters  for  tabulation  and  analysis. 

At  your  request,  we  determined  the  status  and  quality  of  DCS  2000  as  well 
as  the  risks  the  bureau  faces  in  successfully  completing  the  system.  This 
report  summarizes  the  information  presented  at  our  December  14,  1999, 
briefing  to  your  office.  A  copy  of  our  briefing  is  provided  in  appendix  I. 


Results  in  Brief  The  bureau  has  made  considerable  progress  on  DCS  2000.  For  example,  as 

of  January  7,  2000,  DCS  2000  hardware  has  been  installed  at  all  four  data 
capture  centers,  and  21  of  the  system’s  23  planned  application  software 
releases  have  been  completed.  Despite  this  progress,  however,  the  bureau 
faces  a  formidable  challenge  in  delivering  promised  DCS  2000  capabilities 
on  time  because  much  remains  to  be  accomplished  and  little  time  remains 
to  accomplish  it.  In  particular,  the  bureau  has  extended  the  system’s 


'The  data  capture  centers  are  located  in  Baltimore,  Maryland;  Jeffersonville,  Indiana; 
Pomona,  California;  and  Phoenix,  Arizona. 
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schedule  by  4  months  due  largely  to  requirements  changes.  And,  before  its 
revised  completion  date  of  February  25,  2000,  the  bureau  must  complete 
many  important  system  development  and  testing  activities,  including  the 
completion  of  the  final  two  software  releases  as  well  as  system  acceptance, 
site  acceptance,  and  operational  tests.  Furthermore,  the  numbers  of  yet-to- 
be-resolved  defects  for  DCS  2000  have  yet  to  show  a  clear  and  sustained 
downward  trend  that  is  expected  as  a  system  begins  to  mature. 

To  expedite  the  completion  of  DCS  2000,  the  bureau  and  its  development 
contractor  are  following  an  incremental  development  and  deployment 
strategy.  Such  a  strategy  can  save  time  because  it  can  get  a  system  into  the 
hands  of  users  faster  so  that  problems  can  be  identified  sooner  rather  than 
later.  While  the  bureau’s  implementation  of  this  strategy  has  introduced 
considerable  development  and  test  concurrency,  which  increases  the  risk 
of  defects  being  found  and  corrected  independently  and  thus  inconsistent 
system  baselines  being  produced,  this  risk  is  being  mitigated  by  the 
development  contractor  through  effective  management  controls,  such  as 
project  risk  management  and  configuration  management. 

The  bureau  and  its  development  contractor  have  taken  other  steps  to 
ensure  the  successful  delivery  of  DCS  2000.  For  example,  they  have 
implemented  a  formal  risk  management  program  to  address  risks 
proactively,  and  they  have  defined  and  are  following  requirements 
management  processes  to  ensure  that  only  those  changes  that  are  justified 
on  the  basis  of  costs,  benefits,  and  risks  are  approved  and  made.  Also,  they 
have  added  a  test  event  just  prior  to  the  system’s  operational  date  to 
evaluate  the  system’s  capabilities  in  a  true  operational  setting.  Another 
factor  strongly  in  Census’  favor  is  that  its  development  contractor  has  been 
independently  assessed  as  having  highly  effective  software  development 
capabilities  in  such  important  areas  as  software  project  planning,  tracking 
and  oversight,  configuration  management,  software  quality  management, 
and  defect  prevention. 

Nevertheless,  delivering  promised  DCS  2000  capabilities  remains  at  risk 
because  less  than  2  months  remain  before  data  capture  operations  are  to 
begin,  leaving  very  little  room  for  error.  Moreover,  many  important 
development  and  test  activities  remain  that  will  likely  reveal  more  system 
defects  and  thus  compound  an  already  uncertain  system  maturation 
picture.  In  discussing  this  risk  with  DCS  2000  program  officials,  they  agreed 
that  delivering  promised  system  capabilities  on  time  is  a  risk,  and  they 
subsequently  provided  evidence  that  they  have  (1)  designated  this  as  a  high 
risk  under  the  DCS  2000  risk  management  program  and  (2)  defined  and 
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initiated  proactive  steps  to  mitigate  the  risk  and  its  potential  impact  on  the 
program. 

In  commenting  on  a  draft  of  this  report  the  bureau  generally  agreed  with 
our  observations  and  conclusions  and  stated  that  it  appreciated  our 
insights  and  contributions. 


Background 


The  Constitution  requires  a  decennial  census  of  the  population  in  order  to 
reapportion  seats  in  the  House  of  Representatives.  Public  and  private 
decisionmakers  also  use  census  data  on  population  counts  and  social  and 
economic  characteristics  for  a  variety  of  purposes.  For  example,  state  and 
local  redistricting;  allocations  of  government  funding;  and  many  planning 
and  evaluation  activities,  such  as  site  selection  for  new  schools,  market 
research,  and  evaluations  of  local  labor  markets,  rely  on  decennial  census 
data.  In  addition,  the  census  is  the  only  national  source  of  detailed 
population  statistics  for  small  geographic  areas,  such  as  towns  and  school 
districts,  and  for  population  groups,  such  as  Native  Americans. 

Since  1970,  the  bureau  has  used  essentially  the  same  methodology  to 
conduct  the  census.  It  develops  an  address  list  of  the  nation’s  housing  units 
and  delivers  census  forms  to  those  housing  units,  requesting  that 
occupants  mail  back  the  completed  forms.  Most  households  are  sent  a 
short  form  to  complete;  however,  some  are  asked  to  complete  a  long  form.2 
The  bureau  then  hires  temporary  census-takers,  known  as  enumerators,  by 
the  hundreds  of  thousands  to  gather  the  requested  information  for  each 
nonresponding  housing  unit.  For  the  2000  census,  some  households  will 
have  the  option  of  completing  questionnaires  via  a  bureau-operated  toll- 
free  telephone  number  or  a  bureau-operated  Internet  site. 

To  conduct  the  2000  census,  the  bureau  will  rely  on  10  key  systems.  These 
systems  will  enable  the  bureau  to  develop  and  maintain  address  lists,  maps, 
and  geographic  reference  files;  collect  census  data  via  the  Internet;  scan 
and  process  household-completed  paper  forms;  analyze  census  data; 


2The  Census  short  form-three  pages  with  eight  questions-will  be  delivered  to 
approximately  83  percent  of  all  housing  units.  It  will  ask  the  respondent  to  provide 
information  for  up  to  six  household  members  including  information  regarding  name,  age, 
sex,  relationship,  and  race.  The  long  form-40  pages  with  53  questions-will  be  delivered  to 
approximately  17  percent  of  all  housing  units.  For  up  to  six  household  members,  it  will  ask 
the  same  questions  as  the  short  form  as  well  as  questions  on  social,  economic,  financial,  and 
physical  characteristics. 
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recruit  and  support  temporary  workers;  facilitate  follow-up  surveys;  and 
track  costs  and  performance  related  to  taking  the  census.  A  simplified 
diagram  of  the  census  2000  systems  architecture  as  well  as  descriptions  of 
each  system  are  provided  in  appendix  I. 

An  important  system  for  this  census  will  be  DCS  2000.  The  bureau  will  use 
this  system  to  check  in  forms  and  digitally  image  and  optically  read  the 
data  handwritten  onto  census  forms  and  convert  these  data  into  files  that 
will  be  transmitted  to  bureau  headquarters  for  tabulation  and  analysis. 
Specific  DCS  2000  subsystems  include: 

•  The  data  verification  and  receipt  subsystem,  which,  among  other  things, 

(1)  receives  the  paper  census  forms  and  prepares  them  for  imaging  and 

(2)  identifies  respondents  so  the  bureau  can  identify  and  follow  up  with 
nonrespondents. 

•  The  scanning  and  imaging  subsystem,  which  creates  an  electronic  image 
of  the  paper  form. 

•  The  optical  recognition  subsystem,  which  captures  census  data  from  the 
electronic  form  images. 

•  The  keying  subsystem,  which  is  used  to  manually  input  data  that  cannot 
be  satisfactorily  read  from  the  paper  form. 

•  The  data  preparation  function,  which  formats  data  from  the  optical 
recognition  and  keying  subsystems  and  then  sends  the  data  to  Census 
Bureau  headquarters. 

The  bureau  has  contracted  with  Lockheed  Martin-Mission  Systems  to 
develop,  deploy,  and  maintain  DCS  2000.  As  of  November  5,  1999,  the 
bureau  had  obligated  $  127  million  for  this  contract.  The  total  contract  value 
as  of  September  30,  1999,  was  $153  million.  The  bureau  has  also  contracted 
with  TRW  for  space  and  facility  management  and  for  DCS  2000  operation  at 
three  of  the  four  data  capture  centers.3 


Objectives,  Scope,  and 
Methodology 


Our  first  objective  was  to  determine  DCS  2000’s  status  and  quality,  i.e., 
whether  the  system  performs  in  a  way  that  meets  specified  requirements 
and  satisfies  user  needs.  Our  second  objective  was  to  assess  the  risks  the 
bureau  faces  in  successfully  completing  the  system. 


3The  fourth  data  capture  center,  located  at  Census’  National  Processing  Center  in 
Jeffersonville,  Indiana,  will  be  operated  by  the  bureau. 
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To  determine  the  status  and  quality  of  DCS  2000,  we  first  reviewed  system 
development  plans,  project  management  plans,  activity  plans  (test, 
deployment,  etc.),  project  status  reports,  program  management  briefings, 
and  system  functional  requirements  and  specifications.  We  then  compared 
original  to  revised  plans  and  current  plans  to  status  and  test  reports.  We 
also  identified  how  the  bureau  measured  the  maturity  and  quality  of  DCS 
2000  throughout  its  development  life  cycle  and  analyzed  these  and  other 
measures  for  indicators  of  progress  and  system  health,  including  trends  in 
defects  and  the  results  of  testing  activities.  We  observed  the  DCS  2000 
operational  test  at  the  Baltimore  data  capture  center  and  attended  periodic 
DCS  2000  project  review  meetings  between  the  bureau  and  the 
development  contractor. 

To  determine  DCS  2000  risks  and  risk  mitigators,  we  determined  bureau 
and  development  contractor  key  management  processes  (e.g., 
requirements  management,  configuration  management,  project  planning, 
and  project  tracking  and  oversight).  We  analyzed  the  results  of  status  and 
quality  determinations  vis-a-vis  DCS  2000  expectations  and  remaining  steps 
to  complete  the  system.  We  also  analyzed  plans  for  completing  DCS  2000 
and  compared  these  plans  to  generally  accepted  system  engineering 
principles. 

We  interviewed  bureau  and  contractor  officials  throughout  our  review, 
briefed  them  on  the  results  of  our  work,  and  incorporated  updated 
information  they  provided  into  our  briefing.  We  performed  our  work  at  the 
Census  Bureau’s  headquarters  in  Suitland,  Maryland;  the  DCS  2000 
program  office  in  Lanham,  Maryland;  bureau  and  development  contractor 
facilities  in  Bowie,  Maryland;  and  the  Census  Bureau’s  data  capture  center 
in  Baltimore,  Maryland,  from  June  1999  through  January  2000,  in 
accordance  with  generally  accepted  government  auditing  standards.  We 
requested  comments  on  a  draft  of  this  report  from  the  Department  of 
Commerce.  The  Secretary  of  Commerce  provided  us  with  written 
comments,  which  are  presented  in  appendix  II,  and  are  discussed  in  the 
“Agency  Comments  and  Our  Evaluation”  section  of  this  letter. 


Considerable  Progress 
Made  on  DCS  2000  But 
Much  Remains  to  Be 
Done  in  Little  Time 


The  Census  Bureau  has  made  considerable  progress  on  DCS  2000.  For 
example,  as  of  January  7,  2000,  21  of  23  planned  software  releases  had  been 
completed  and  6  of  10  major  test  events  had  been  completed.  Further, 
according  to  the  bureau,  all  DCS  2000  hardware  has  been  installed  at  all 
sites.  As  of  January  7,  2000,  the  bureau  was  reporting  that  remaining  DCS 
2000  tasks  were  on  target  with  revised  expectations. 
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Nevertheless,  DCS  2000’s  expected  completion  is  much  later  than  earlier 
planned  and,  in  fact,  perilously  close  to  the  date  when  census  2000  is  to 
begin.  In  February  1999,  the  bureau  expected  to  complete  DCS  2000  by 
October  15,  1999.  However,  the  scheduled  completion  date  has  since 
changed  to  February  25,  2000,  which  is  an  extension  of  over  4  months  and 
is  less  than  2  weeks  before  data  capture  operations  are  to  begin  on 
March  6,  2000.  Under  normal  circumstances,  completion  deadlines  can  be 
changed  and  some  schedule  slippage  can  be  tolerated.  However,  the 
demands  of  the  decennial  census  necessitate  that  remaining  development, 
testing,  and  deployment  activities  be  completed  in  less  than  2  months. 

DCS  2000  program  officials  told  us  that  this  extension  is  due  largely  to 
system  requirement  changes.  Specifically,  from  May  15,  1999,  through 
October  30,  1999,  63  requirements  changes  were  initiated  for  DCS  2000. 
And,  in  October  and  November  1999,  two  new  software  releases  were 
added  to  DCS  2000-one  slated  for  January  19,  2000,  and  one  for 
February  11,  2000.  The  bureau  has  no  plans  for  additional  requirements 
changes.  However,  if  the  results  of  ongoing  and  planned  operational  tests 
show  that  DCS  2000  does  not  adequately  support  production  environment 
data  capture  operations,  then  more  changes  are  possible. 

In  addition  to  the  development  and  deployment  of  final  software  releases, 
significant  testing  activities — including  system,  site,  and  operational  tests4- 
remain  to  be  completed  before  February  25,  2000.  These  activities  are 
illustrated  in  figure  1. 


'’DCS  2000  is  currently  undergoing  three  types  of  major  test  activities:  (1)  system  acceptance 
testing,  which  assesses  whether  the  system  performs  according  to  specifications,  (2)  site 
acceptance  testing,  which  assesses  whether  the  system  performs  correctly  at  its 
deployment  site,  and  (3)  operational  testing,  which  assesses  whether  the  system  performs 
as  intended  when  operated  on-site  by  those  expected  to  use  it. 
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Figure  1 :  Remaining  DCS  2000  Activities 
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Results  from  early  site  acceptance  tests  showed  that  not  all  performance 
targets  were  achieved;  however,  the  tests  were  reported  as  being 
successful  either  because  the  problems  were  corrected  and  retested  or 
because  they  were  scheduled  for  correcting  and  retesting  as  part  of  a  future 
test  event.  More  recent  results  show  that  key  performance  measures  are 
being  met.  For  example,  as  shown  in  figures  2  and  3,  results  from  “cluster” 
accuracy  tests — which  were  performed  to  demonstrate  the  ability  of  each 
individual  configuration  (known  as  “clusters”)  of  scanners,  workstations, 
and  servers  to  scan  census  forms  and  recognize  data  on  those  forms — were 
within  expected  tolerances  for  errors. 
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Figure  2:  Results  of  Cluster  Accuracy  Tests  for  Optical  Mark  Recognition 


Figure  3:  Results  of  Cluster  Accuracy  Tests  for  Optical  Character  Recognition 
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|  =  Tick  marks  show  individual  error  rate  measurements  at  each  location, 
i  =  White  area  shows  acceptable  range  of  error  rate  (in  percentage). 
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Similarly,  results  from  the  operational  test  conducted  at  the  Baltimore, 
Maryland,  data  capture  center  identified  problems  that  have  since  been 
reported  as  fixed  and  scheduled  to  be  retested  as  part  of  a  future 
operational  test.  An  additional  operational  test  was  conducted  at  the 
Baltimore  data  capture  center  by  an  independent  research  organization  to 
assess  DCS  2000’s  accuracy.  Results  from  this  test  showed  that  DCS  2000 
exceeded  accuracy  goals  for  both  short  and  long  Census  forms. 

Despite  these  reported  successes,  DCS  2000’s  readiness  is  still  uncertain 
due  to  the  fact  that  (1)  the  numbers  of  yet-to-be-resolved  system  defects 
have  yet  to  show  a  clear  and  sustained  downward  trend  expected  as  a 
system  begins  to  mature  and  (2)  the  time  left  between  the  conclusion  of 
test  events  and  the  day  that  DCS  2000  must  be  operational  is  less  than  the 
average  time  it  has  been  taking  to  resolve  defects.  Defects  are  system 
problems  (both  hardware  and  software)  that  require  a  resolution  (referred 
to  by  Lockheed  Martin  as  Program  Trouble  Reports,  or  PTRs).  They  can  be 
attributed  to  a  failure  to  meet  system  specifications  or  to  a  requirement 
that  was  not  previously  anticipated.  As  figures  4  and  5  further  illustrate,  we 
found  that  the  number  of  unresolved  PTRs  each  week  during  the  last  12 
months  does  not  show  a  maturing  trend,  including  those  defects 
categorized  as  severe.5  In  addition,  as  mentioned  earlier,  many  key  test 
events,  which  are  major  producers  of  PTRs,  have  yet  to  be  conducted  for 
DCS  2000.  Lastly,  the  time  left  between  conclusion  of  the  last  test  event, 
scheduled  for  February  25,  2000,  and  the  day  that  DCS  2000  must  be 
operational  is  9  days,  whereas  it  has  been  taking  an  average  of  16  days  to 
resolve  all  PTRs  over  the  last  6  months  and  10  days  for  severe  PTRs. 


q’wo  categories  of  severe  defects  include  emergency- a  problem  with  no 
deliverable/procedural  workaround  available,  which  blocks  operational  use  of  the  product 
or  completion  of  all  or  some  test  cases  or  test  phases-and  critical-a  problem  with  no 
deliverable/procedural  workaround  available,  which  significantly  degrades  operational  use 
or  which  blocks  a  top-level  test  case  or  test  phase  from  completion. 
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Figure  4:  Trends  in  Defects — All  Unresolved  PTRs  by  Week 
#  of  PTRs 


-  -  -  Unresolved  PTRs  by  week 


•8-week  moving  average 
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Figure  5:  Trends  in  Defects — Unresolved  Severe  PTRs  by  Week 
#  of  PTRs 


-  -  -  Unresolved  PTRs  by  week —8-week  moving  average 


Risk  Mitigation  Steps 
Are  Being  Taken  to 
Address  DCS  2000 
Risks 


The  bureau  and  its  contractors  are  employing  measures  to  expedite  the 
completion  of  DCS  2000  and  to  minimize  risks  related  to  its  tight  schedule 
and  other  aspects  of  development  and  deployment.  The  bureau  and 
Lockheed  Martin  are  following  an  incremental  development  and 
deployment  strategy.  Such  a  strategy  is  recognized  as  a  best  practice  for 
large,  complex  systems  by  leading  public  and  private  organizations 
because  it  can  get  a  system  into  the  hands  of  users  faster  so  that  problems 
can  be  identified  sooner  rather  than  later. 
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This  approach  needs  to  be  carefully  managed  because  it  has  introduced 
considerable  development  and  test  concurrency  in  the  DCS  2000  schedule. 
Such  concurrency  can  increase  demands  for  test  resources  as  well  as  the 
possibility  of  defects  being  found  and  corrected  independently,  resulting  in 
more  than  one  baseline  test  configuration.  Moreover,  if  defects  surface 
during  concurrent  testing  that  are  severe  enough  to  require  all  affected  test 
activities  to  be  stopped  until  the  baseline  is  fixed,  the  deployment  schedule 
could  be  delayed.  We  found  that  these  risks  are  being  managed  by 
Lockheed  Martin  through  the  use  of  multiple  test  labs  and  effective 
configuration  management  processes. 

The  bureau  and  Lockheed  Martin  are  also  jointly  managing  DCS  2000  risks 
through  a  formal  risk  management  program  that  includes  proactive 
identification  of  risks  and  analyses  of  their  impacts  and  probabilities, 
definition  of  risk  mitigation  plans,  and  tracking  and  reporting  of  the  plans’ 
implementation  and  effectiveness.  Such  programs  are  also  recognized  as  a 
key  practice  in  government  and  the  private  sector.  In  following  the  risk 
mitigation  program,  the  bureau  and  Lockheed  Martin  have  already  acted  to 
resolve  several  risks  related  to  DCS  2000  site  staffing,  daily  system 
maintenance,  and  technical  support.  For  example,  they  have  strengthened 
training  programs  for  on-site  DCS  2000  personnel,  defined  technical 
support  roles  and  responsibilities,  and  increased  central  technical  support. 

More  important,  the  bureau  and  Lockheed  Martin  have  also  identified  the 
tight  development  schedule  as  a  risk  and  taken  measures  to  reduce  the 
risk.  For  example,  they  have 

•  added  staff  to  manage  PTR  progress; 

•  begun  holding  regular  PTR  status  meetings  with  sites  and  other  support 
areas  to  review  resolution  progress; 

•  developed  a  rapid  response  process  that  requires  a  fix  or  workaround 
within  24  hours; 

•  established  an  Engineering  Review  Board  to  review  all  change  requests 
to  ensure  that  only  changes  that  are  justified  on  the  basis  of  costs, 
benefits,  and  risks  are  approved  and  made;  and 

•  implemented  processes  to  ensure  that  all  releases  with  new 
functionality  are  tested  during  an  operational  evaluation. 

Furthermore,  Census  and  Lockheed  Martin  have  added  another  test  to  the 
DCS  2000  schedule  to  demonstrate  simultaneous  operation  of  all  four  data 
capture  centers.  The  test,  planned  for  February  22  through  February  25, 
will  include  assessments  of  DCS  2000’s  ability  to  process  workload 
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equivalent  to  that  expected  during  actual  data  capture  operations,  and  it 
will  include  operational  testing  of  the  final  DCS  2000  software  releases. 

Another  important  risk  mitigating  factor  is  that  Lockheed  Martin-Mission 
Systems  has  been  independently  assessed  as  having  highly  effective 
software  development  capabilities  in  important  areas  such  as  software 
project  planning,  tracking  and  oversight,  and  configuration  management. 
This  assessment  was  performed  using  models  and  methods  developed  by 
Carnegie  Mellon  University’s  Software  Engineering  Institute  (SEI),  which  is 
recognized  for  its  expertise  in  software  processes.  With  DCS  2000  as  one  of 
three  systems  included  in  the  evaluation,  Lockheed  Martin-Mission 
Systems  was  rated  as  a  “Level  5”  (on  a  scale  of  1  to  5),  meaning  that  all 
projects  use  standard  processes  for  developing  and  maintaining  software 
and  that  such  processes  are  continually  improved  through  quantitative 
feedback  and  from  piloting  innovative  ideas  and  technologies.  Having  such 
capability  should  help  to  reduce  risks  related  not  only  to  developmental 
activities  such  as  project  planning  and  oversight,  but  all  remaining  DCS 
2000  development,  testing,  and  deployment  activities. 


Conclusions 


The  Census  Bureau  has  made  considerable  progress  in  acquiring  and 
deploying  DCS  2000;  however,  its  prospects  for  completing  this  system  on 
time  are  still  mixed.  In  particular,  significant  system  functionality  remains 
to  be  developed,  acceptance  tested,  deployed,  and  site  tested.  Further,  a 
key  measure  of  system  quality-trends  in  unfixed  severe  defects-is  not  yet 
moving  in  a  direction  consistent  with  that  of  a  maturing  system.  Given  that 
a  number  of  significant  test  events  have  yet  to  occur  and  these  are  likely  to 
uncover  more  defects,  the  chances  of  the  defect  trend  moving  in  a 
favorable  direction  soon  are  uncertain.  Moreover,  given  that  the  average 
time  to  fix  severe  defects  (about  10  days)  over  the  last  6  months  would 
consume  the  time  between  the  conclusion  of  later  test  events  and  the  day 
that  DCS  2000  must  be  operational,  the  chances  of  all  specified  capabilities 
being  ready  on  time  are  reduced. 

At  the  same  time,  results  from  system-level  tests  performed  so  far  are 
showing  that  key  DCS  2000  performance  targets  are  being  met. 
Additionally,  to  meet  the  challenge  of  a  very  demanding  and  fault-intolerant 
schedule,  the  bureau  has  been  following  a  plan  that  provides  for 
development  and  testing  of  DCS  2000  in  a  series  of  increments.  This 
strategy  could  save  time  by  allowing  early  system  capability  to  get  into  the 
hands  of  end  users  faster.  Lastly,  the  development  contractor  has  been 
following  effective  processes  for  risk  management,  project  planning, 
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tracking  and  oversight,  configuration  management,  software  quality 
management,  and  defect  prevention-all  critical  to  the  successful 
completion  of  DCS  2000. 

Nevertheless,  the  schedule  challenge  facing  the  bureau  on  DCS  2000, 
particularly  in  light  of  where  the  system  currently  stands  and  what  remains 
to  be  accomplished,  is  formidable.  Both  the  bureau  and  development 
contractor  agree  and,  accordingly,  have  initiated  proactive  and  appropriate 
risk  management  steps,  such  as  strengthening  oversight  over  PTR  progress 
and  requirements  changes.  While  these  steps  do  not  guarantee  success, 
they  should  mitigate  the  risk  and  its  potential  impact  on  the  program. 


In  commenting  on  a  draft  of  this  report,  the  bureau  stated  that  it  generally 
agreed  with  our  observations  and  conclusions.  Additionally,  the  bureau 
stated  that  it  shared  our  awareness  of  the  challenges  the  bureau  faces  in 
delivering  promised  DCS  2000  capabilities  on  time,  and  that  it  appreciated 
our  insights  and  contributions. 

The  bureau  also  offered  two  additional  comments.  First,  it  stated  that  one 
sentence  in  the  draft  report  implies  that  all  of  the  DCS  2000  system  must  be 
developed,  tested,  and  deployed  in  2  months  while,  in  fact,  the  system  is  in 
the  final  stages  of  testing,  and  only  a  small  portion  of  development  remains 
to  be  completed  before  becoming  fully  operational.  It  was  not  our  intent  to 
imply  this  and  we  have  accordingly  modified  the  sentence. 

Second,  the  bureau  stated  that  the  16  days  that  the  bureau  has  averaged 
over  the  last  6  months  to  resolve  all  PTRs  that  we  cite  in  the  draft  report  is 
not  a  true  representation  of  the  time  it  has  been  recently  taking  to  resolve 
emergency  PTRs,  which  according  to  the  bureau  was  2  days  for  7  recent 
emergency  PTRs.  We  did  not  modify  our  report  in  response  to  this 
comment.  The  report  does  not  state  that  the  16-day  average  time  needed 
over  the  last  6  months  (which  covered  June  through  November  1999)  to 
resolve  all  PTRs  is  the  operative  indicator  of  the  time  to  resolve  emergency 
PTRs,  which  is  the  most  severe  category  of  PTRs.  Rather,  it  cites  the  time 
necessary  to  resolve  severe  PTRs — which  was  10  days — as  the  operative 
indicator.  This  category  represents  the  average  time  that  the  bureau’s  own 
data  show  it  has  taken  over  the  last  6  months  to  resolve  both  emergency 
and  critical  PTRs.  We  focused  on  both  emergency  and  critical  because  both 
types  of  PTRs,  according  to  the  Bureau’s  own  definition  of  them,  will  have 
to  be  resolved  before  March  6,  2000,  if  DCS  2000  is  to  operate  as  intended. 


Agency  Comments  and 
Our  Evaluation 
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We  are  sending  copies  of  this  report  to  the  Honorable  William  M.  Daley, 
Secretary  of  Commerce;  the  Honorable  Kenneth  Prewitt,  Director  of  the 
U.S.  Census  Bureau;  the  Honorable  Jacob  J.  Lew,  Director  of  the  Office  of 
Management  and  Budget;  and  other  interested  parties.  Copies  will  be  made 
available  to  others  upon  request. 

If  you  have  any  questions  on  matters  discussed  in  this  letter,  please  contact 
me  at  (202)  512-6240.  Other  key  contributors  to  this  report  include  Mark 
Bird,  Cristina  Chaplain,  Garry  Durfey,  Richard  Hung,  and  Aaron  Thorne. 


Randolph  C.  Hite 
Associate  Director 

Governmentwide  and  Defense  Information  Systems 
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Briefing  to  the 


Subcommittee  on  the  Census,  Committee  on 
Government  Reform,  House  of  Representatives 

Decennial  Census:  Data  Capture  System 
2000  Progress  and  Risks 

December  14,  1999 


A 
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GAO  Briefing  Overview 


•  Introduction 

•  Objectives 

•  Results  in  Brief 

•  Background 

•  Audit  Results:  DCS  2000  Status  and  Quality 

•  Audit  Results:  DCS  2000  Risks/Risk  Mitigators 

•  Conclusions 

•  Appendixes 

-  Scope  and  Methodology 

-  Description  of  Other  Decennial  Census  Systems 
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GAO  Introduction 


•  Required  by  the  Constitution,  our  nation’s  decennial 
census  is  a  landmark  event  whose  results  are 
instrumental  in  decisions  affecting  the  lives  of  every  U.S. 
citizen. 

•  The  2000  decennial  census  is  the  largest  peacetime 
mobilization  event  in  our  nation’s  history,  entailing  the 
staffing  of  about  1 .35  million  temporary  positions  to 
collect  and  capture  1 .5  billion  pages  of  data  from  about 
119  million  households. 

•  Information  technology  is  the  critical  enabler  for 
successfully  accomplishing  this  mammoth  and  historical 
event. 
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GAO  Objectives 


•  Because  the  Data  Capture  System  (DCS)  2000  is 
viewed  as  an  important  system  in  accomplishing  the 
2000  census,  the  Chairman  and  Ranking  Minority 
Member  of  the  Subcommittee  asked  GAO  to  determine 
the  following. 

-  What  is  the  status  and  quality  of  DCS  2000? 

-  What  risks  does  the  bureau  face  in  successfully 
completing  DCS  2000? 

•  The  scope  and  methodology  of  our  review  is  described 
in  appendix  I. 
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GAO  Results  in  Brief 


Status  and  Quality 

•  Since  February  1 999  and  largely  because  of  system 
requirements  changes,  the  DCS  2000  completion  date  has 
been  extended  by  over  4  months  (meaning  that  the  last 
system  release  is  now  scheduled  for  deployment  less  than 
4  weeks  before  the  system’s  immovable  March  6,  2000, 
deadline). 

•  Changes  to  system  requirements  continue  to  occur,  with 
more  activities  being  added  to  an  already  challenging 
schedule,  but  change  management  controls  are  in  place 
and  functioning. 
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GAO  Results  in  Brief  (cont.) 


•  The  bureau  reports  that  DCS  2000  is  meeting  the  revised 
schedule  and  that  it  is  meeting  cost  expectations. 

•  Trends  in  yet-to-be  resolved  system  defects  are  not 
consistent  with  those  of  a  maturing  system. 

•  Five  of  the  1 0  scheduled  system-level  test  events  have 
been  completed,  and,  while  early  test  event  results  show 
that  not  all  performance  measures  were  satisfied,  more 
recent  test  event  results  show  that  key  performance 
measures  are  being  met. 
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GAO  Results  in  Brief  (cont.) 


Risks 

•  Important  system  development,  test,  and  deployment 
activities  remain  that  will  likely  generate  more  system 
defects,  and  the  current  schedule  will  not  accommodate 
slippage. 

•  Concurrent  system-level  test  activities  increase  contention 
for  test  resources  and  complicate  control  of  test  baseline 
system  configurations. 

Risk  Mitigators 

•  Incremental  development  and  deployment  gets  system 
into  hands  of  users  faster. 
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GAO  Results  in  Brief  (cont.) 


•  Formal  risk  management  program  is  in  place  to  address 
risks  proactively,  including  steps  to  mitigate  the  possibility 
of  not  completing  remaining  system  development  activities 
on  time. 

•  An  additional  test  event  to  exercise  all  planned  system 
capabilities  in  an  operational  environment  is  to  occur  just 
prior  to  the  March  6,  2000,  deadline. 

•  Development  contractor’s  software  process  capabilities 
are  effective,  including  processes  for  configuration 
management. 
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GAO  Results  in  Brief  (cont.) 


Bottom  Line  on  DCS  2000  Status.  Quality,  and  Risks 

•  Considerable  progress  has  been  made  on  DCS  2000  and 
important  risk  mitigators  exist. 

•  Nevertheless,  delivering  promised  system  capabilities  on 
time  is  a  high  risk  because: 

-very  little  time  remains  (3  months),  and  the  deadline  is 
immutable; 

-  many  important  development  and  test  activities  remain; 

-  new  requirements  continue  to  be  added; 

-  trends  in  system  defects  do  not  yet  demonstrate  system 
maturity;  and 
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GAO  Results  in  Brief  (cont.) 


-  more  defects  are  likely  to  surface  during  later  test 
events. 

•  On  December  7, 1999,  we  briefed  DCS  2000  program 
officials  on  the  results  of  our  work  and  they  provided 
updated  information  that  has  been  incorporated  into  this 
briefing  as  appropriate.  The  officials  agreed  that 
delivering  promised  system  capabilities  on  time  is  a  high 
risk,  and  they  subsequently  provided  evidence  that 
(1)  they  have  designated  this  as  a  high  risk  under  the  DCS 
2000  risk  management  program  and  (2)  they  have  defined 
and  initiated  proactive  steps  to  minimize  the  probability  of 
the  risk  occurring  and  the  impact  on  the  program  if  it  does. 
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GAO  Background 

Decennial  Census  Serves  Many  Purposes 


•  Public  and  private  decisionmakers  use  census  population 
and  socioeconomic  data  for  various  purposes.  For 
example, 

-  the  bureau  will  deliver  to  the  President  the  population 
counts  needed  to  determine  each  state’s  number  of 
seats  in  the  House  of  Representatives; 

-  each  state  will  use  tabulations  in  its  individual 
redistricting  plan; 

-  federal  agencies  can  use  census  data  to  evaluate  the 
effectiveness  of  established  programs  and  identify 
geographic  areas  needing  attention; 


11 
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GAO  Background 

Decennial  Census  Serves  Many  Purposes  (cont.) 


-  local  governments  will  use  census  data  to  target 
delivery  of  services,  such  as  schools  and  public 
transportation;  and 

-  businesses  will  use  census  data  to  target  new  services 
and  products  and  tailor  existing  ones  to  demographic 
changes. 
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GAO  Background 

Overview  of  How  2000  Census  Data  Will  Be  Captured 


•  The  primary  method  for  gathering  census  data  is  through 
paper  questionnaires  mailed  to  households  that  the 
occupants  are  to  complete  and  mail  back.  Paper 
questionnaires  will  also  be  available  in  public  locations  for 
people  to  fill  out  and  mail  back. 

•  Some  households  will  also  have  the  option  of  responding 
via  a  toll-free  telephone  number  or  the  Internet,  rather  than 
completing  and  mailing  back  paper  forms. 

•  Bureau  procedure  calls  for  nonresponding  households  to 
be  visited  by  an  enumerator,  who  will  fill  out  a  paper  form 
for  each  household. 
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GAO  Background 

Overview  of  Systems  Supporting  2000  Census 

•  The  bureau  relies  on  10  key  systems  to  execute  the  2000 
decennial  census,  including  DCS  2000.  The  other  nine 
systems  are  listed  below  and  described  in  appendix  II. 

-  Geographic  Support  System  (GSS) 

-  Pre-Appointment  Management  System/Automated 
Decennial  Administrative  Management  System 
(PAMS/ADAMS) 

-  Operations  Control  System  (OCS  2000) 

-  Management  Information  System  (MIS  2000) 

-  Telephone  Questionnaire  Assistance  and  Coverage 
Edit  Follow-Up  (TQA/CEFU) 
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GAO  Background 

Overview  of  Systems  Supporting  2000  Census  (cont.) 


-  Internet  Data  Collection/Internet  Questionnaire 
Assistance  (IDC/IQA) 

-  Accuracy  and  Coverage  Evaluation  System  (ACE) 

-  Headquarters  (HQ)  Processing 

-  Data  Access  and  Dissemination  System  (DADS) 
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GAO  Background 

Simplified  Diagram  of  Decennial  Census  Systems  Architecture 
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GAO  Background 

Simplified  Diagram  of  DCS  2000  Functional 
Architecture 
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GAO  Background 

DCS  2000  Technical  Architecture  Overview 


•  DCS  2000  consists  primarily  of  mail  sorters,  document 
scanners,  workstations,  and  servers  running  in  a 
Windows  NT-based  environment. 

•  DCS  2000  software  is  commercial  off-the-shelf  (COTS); 
new  software  development  amounts  to  about  85,000  lines 
of  code  written  in  C,  C++,  and  Perl. 

•  New  system  development  is  for  modifying  COTS,  creating 
new  applications,  and  integrating  components. 

•  DCS  2000  software  is  being  developed  in  23  incremental 
releases. 

•  Scanners,  workstations,  and  servers  are  configured  into 
clusters. 
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GAO  Background 

DCS  2000  Technical  Architecture  Overview  (cont.) 


•  Each  data  capture  center  (DCC)  is  configured  with  the 
number  of  clusters  needed  to  support  expected  workload. 

-  Baltimore:  15  clusters 

-  Jeffersonville:  1 0  clusters 

-  Pomona:  14  clusters 

-  Phoenix:  15  clusters 
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GAO  Background 

DCS  2000  Technical  Architecture  Overview  (cont.) 
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GAO  Background 

DCS  2000  Project  Structure 


•  The  bureau  has  contracted  with  Lockheed  Martin-Mission 
Systems  for  DCS  2000  development,  deployment,  and 
maintenance. 

•  For  three  of  four  DCCs,  the  bureau  has  contracted  with 
TRW  for  space  and  facility  management  and  for  DCS 
2000  operation.  The  bureau  plans  to  operate  DCS  2000 
at  the  fourth  DCC  (the  bureau’s  National  Processing 
Center). 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 
(cont.) 


1 .  Since  February  1 999  and  largely  because  of  system 
requirements  changes,  the  DCS  2000  completion  date 
has  been  extended  by  over  4  months  (meaning  that  the 
last  system  release  is  now  scheduled  for  deployment 
just  4  weeks  before  the  system’s  immovable  March  6, 
2000,  deadline). 

2.  Changes  to  system  requirements  continue  to  occur,  with 
more  activities  being  added  to  an  already  challenging 
schedule,  but  change  management  controls  are  in  place 
and  functioning. 
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3.  The  bureau  reports  that  DCS  2000  is  meeting  the 
revised  schedule  and  that  it  is  meeting  cost 
expectations. 

4.  Trends  in  yet-to-be  resolved  system  defects  are  not 
consistent  with  those  of  a  maturing  system. 

5.  Five  of  the  10  scheduled  system-level  test  events  have 
been  completed  and  while  early  test  event  results  show 
that  not  all  performance  measures  were  satisfied,  more 
recent  test  event  results  show  that  key  performance 
measures  are  being  met. 
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1 .  Scheduled  Completion  Has  Changed 


•  In  February  1999,  the  scheduled  date  for  completing 
DCS  2000  was  October  15,  1999.  The  scheduled  date 
has  since  changed  to  February  25,  2000,  an  extension  of 
over  4  months  and  less  than  2  weeks  before  data  capture 
operations  are  to  begin. 

•  Program  officials  told  us  that  this  extension  is  largely  due 
to  system  requirements  changes. 

•  In  October/November  1999,  major  activities  were  added 
to  the  end  of  the  schedule. 

-  Two  new  software  releases  were  added,  one  on 
January  19,  2000,  and  one  on  February  1 1 , 2000. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

1 .  Scheduled  Completion  Has  Changed  (cont.) 


-  A  new  operational  test  at  the  four  DCCs  was  added  for 
February  22-25,  2000.  This  additional  event  is  planned 
to  demonstrate  simultaneous  operation  of  all  four 
DCCs,  including  DCS  2000’s  capability  to  process  the 
workload  expected  during  data  capture  operation. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

2.  System  Requirements  Changing  But  Controlled 


•  Changes  to  system  requirements,  particularly  those  made 
late  in  the  development  cycle,  can  impact  the  delivery  of 
promised  system  capabilities  on  time  and  within  budget. 
Accordingly,  changes  to  requirements  must  be  controlled 
to  ensure  that  only  changes  that  are  justified  on  the  basis 
of  costs,  benefits,  and  risks  are  approved  and  made. 

•  For  DCS  2000,  63  requirements  changes  were  initiated 
between  May  15,  1999  and  October  30,  1999. 

•  We  reviewed  the  bureau’s  management  process  for 
controlling  requirements  changes  and  analyzed  selected 
DCS  2000  requirements  changes  to  ensure  they  were 
managed  in  accordance  with  this  management  process. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

2.  System  Requirements  Changing  But  Controlled  (cont.) 


•  We  reviewed  five  requirements  change  requests. 


Change 

control 

#006 

Submitted 

12/15/98 

After  data  have  been  captured,  review  all 
forms  to  ensure  that  headquarters  has 
received  the  information. 

Reviewed 
2/1 0/99 

Approved 

Change 

control 

#028 

Submitted 

3/1/99 

Modify  the  audit  resolution  process  so  that 
the  system  better  handles  images  that  fail  the 
audit  process. 

Reviewed 

1 2/1 0/98 

Approved 

Change 

control 

#047 

Submitted 

4/29/99 

Create  three  sets  of  CD-ROMs  of 
questionnaire  images  on  a  daily  basis  at 
each  site  for  the  ACE  system. 

Reviewed 

7/23/99 

Approved 

Change 

control 

#070 

Submitted 

8/2/99 

Use  remote  copy  protocol  (RCP)  to  transfer 
the  Title  1 3  acknowledgement  files  from 
Census  headquarters  to  the  data  capture 
centers. 

Reviewed 

8/31/99 

Approved 

Change 

control 

#75 

Submitted 

8/12/99 

Add  additional  checkout  reporting  data,  as 
requested  by  TRW. 

Reviewed 

9/17/99 

Disapproved 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

2.  System  Requirements  Changing  But  Controlled  (cont.) 


•  We  selected  these  five  change  requests  using  the 
following  criteria:  (1)  significant  in  terms  of  cost  or  impact, 

(2)  representative  of  other  change  requests,  and 

(3)  demonstrated  request  approval  and  disapproval. 
Bureau  officials  agreed  that  these  requests  met  these 
criteria. 

•  We  found  that  the  bureau’s  change  control  management 
process  provided  for  reviews  of  costs,  benefits,  alternative 
solutions,  and  risks. 

•  We  also  found  that  the  five  requirements  changes  that  we 
reviewed  were  assessed  and  approved/disapproved  in 
accordance  with  the  bureau’s  management  process. 
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GAO  Audit  Results:  Status  and  Quality 

3.  Progress  Against  Current  Schedule  and  Cost  Baselines 


KEY  EVENTS _ 

System  acceptance  test 


1999  _ _ _ _ | _ _ 2000  _ 

May  I  Jun  I  Jul  I  Aug  |  Sep  |  Oct  |  Nov  |  Dec  Jan  |  Feb  |  Mar 


Release  18A 
Release  18B 


Release  18C 


Site  acceptance  test  at  Baltimore 
Site  acceptance  test  at  Pomona 
Site  acceptance  test  at  Phoenix 
Baltimore  operational  test 
Pomona  operational  test 


09/20/99  EH  10/08/99 


11/01/99  11/19/99 


Planned  progress 


Legend 


|  Actual  progress 


Planned  milestone 


Actual  milestone  ♦ 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

3.  Progress  Against  Current  Schedule  and  Cost  Baselines 

•  All  DCS  2000  hardware  reported  as  installed  at  all  sites. 

•  Reported  DCS  2000  contract  costs  are  as  follows: 

-  Obligated:  $127  million  reported  (as  of  November  5, 

1999) and 

-  Contract  value:  $153  million  (as  of  September  30, 

1999). 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects  Raise  Questions  About  System 

•  One  measure  of  system  maturation  and  quality  is  trends 
in  defects. 

•  Defects  are  system  problems  (hardware  and  software) 
that  require  a  resolution.  A  defect  can  be  due  to  a  failure 
to  meet  the  system  specifications,  or  it  can  occur  because 
a  requirement  is  identified  that  was  not  in  the  system 
specification. 

•  Effective  system  engineering  requires  that  defects  be 
formally  managed  (identified,  documented,  and  resolved). 
To  facilitate  defect  management,  defects  are  categorized 
by  severity  (impact)  and  age  (time  unresolved). 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects  Raise  Questions  About  Maturity 

ib^ ________ 

•  Lockheed  Martin  manages  DCS  2000  defects,  called 
program/trouble  reports  (PTRs),  according  to  five  severity 
categories. 


Emergency 

A  problem  with  no  deliverable/procedural  workaround 
available,  which  blocks  operational  use  of  the  product  or 
completion  of  all  or  some  test  cases  or  test  phases. 

Critical 

A  problem  with  no  deliverable/procedural  workaround 
available  which  significantly  degrades  operational  use 
or  which  blocks  a  top-level  test  case  or  test  phase  from 
completion. 

Important 

A  problem  with  a  deliverable/procedural  workaround 
available  (noncritical  failed  steps  in  a  test  case  or  test 
phase  would  fall  here). 

Medium 

A  problem  discovered  through  normal  use,  which  is  not 
covered  directly  by  a  test  case  or  test  phase. 

Low 

A  problem  that  needs  to  be  fixed  but  otherwise  does  not 
block  development  or  testing  activities. 

Page  49 


GAO/AIMD-OO-61  2000  Census  Data  Capture  System 


Appendix  I 

Briefing  on  the  Decennial  Census 


GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects  Raise  Questions  About  Maturity 

•  PTRs  are  open  during  the  time  the  software  development 
organization  is  designing  and  implementing  a  fix  or 
change.  After  the  development  organization  has 
implemented  and  tested  the  fix  or  change  and  submitted  it 
for  system  integration  testing,  the  PTR  is  designated  fixed. 
PTRs  are  closed  after  the  fix  or  change  has  successfully 
completed  system  integration  testing. 

•  We  analyzed  trends  in  DCS  2000  PTRs  during  the  last  12 
months. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects--AII  PTRs  Opened  by  Week 


#  of  PTRs 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects-Emergency/Critical  PTRs 

#  of  PTRs  opened 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects--AII  Unresolved  PTRs  by  Week 


36 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects-Unresolved  Critical/Emergency  PTRs 

#of  PTRs 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects-Average  Age  of  All  PTRs 


Days 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects-Average  Age  of  Emergency/Critical 
PTRs 

Days 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

4.  Trends  in  Defects-Composite  View 


•  Data  on  the  number  of  PTRs  opened  and  unresolved  each 
week  do  not  show  a  maturing  trend. 

•  Test  events  are  major  producers  of  PTRs  and  many  key 
test  events  have  yet  to  occur. 

•  Average  time  in  the  last  6  months  to  resolve  (fix)  all  PTRs 
was  about  1 6  days;  emergency  and  critical  PTRs  was 
about  1 0  days.  A  key  test  event  will  conclude  less  than  2 
weeks  before  the  system  must  be  operational. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 
_ Successful 


•  Acceptance  testing  is  intended  to  show  that  a  system 
correctly  performs  according  to  the  system  specification, 
i.e.,  performs  specified  functions  within  defined 
performance  parameters.  Acceptance  testing  is  generally 
conducted  by  the  developing  organization,  with  the 
acquiring  organization’s  participation. 

•  Operational  testing  is  intended  to  demonstrate  that  a 
system  performs  as  intended  when  operated  on-site  by 
those  expected  to  use  it. 

•  DCS  2000  is  undergoing  three  types  of  major  test 
activities. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 

-  System  Acceptance  Testing  (SAT)  is  to  be  performed 
on  the  incremental  releases  of  DCS  2000.  As  of 
November  3,  1999,  309  of  DCS  2000’s  405  top-level 
requirements  had  been  tested  (about  76  percent). 

-  Site  Acceptance  Testing  (SiteAT)  is  to  be  performed  by 
Lockheed  Martin,  with  government  witnesses,  at  each 
of  the  four  DCCs  on  the  system  functionality  released 
as  of  the  date  of  the  scheduled  SiteAT. 

-  Baltimore,  Pomona,  and  Phoenix  SiteAT  are 
completed. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 

-  According  to  Lockheed  Martin’s  test  results  report  for 
Baltimore’s  SiteAT,  the  test  disclosed  various  system 
problems;  however,  the  test  was  reported  successful 
because  the  problems  were  either  (1)  corrected  and 
retested  or  (2)  scheduled  for  correction  and  retesting  as 
part  of  a  future  test  event.  The  following  are  examples  of 
tests  and  results  from  Baltimore’s  SiteAT. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 
5.  Test  Events  Remain;  Completed  Events  Reported 
_ Successful  (cont.) _ 


Test  Name  and  Description 

Test  Results 

Cluster  Accuracy  :  This  test’s  purpose  was 
to  demonstrate  each  individual  cluster’s 
ability  to  scan  a  small  number  of  census 
forms  and  recognize  data  on  those  forms 
within  specified  tolerances.  The  expected 
tolerance  for  optical  mark  recognition 
(OMR)  errors  was  0%  to  1 .5%  and  the 
expected  range  for  optical  character 
recognition  (OCR)  errors  was  0%  to  2.0%. 

The  test  was  reported  successful.  For 
short  forms  the  OMR  error  rate  ranged 
from  .06%  to  .66%  and  the  OCR  error 
rate  ranged  from  .07%  to  .14%. 

Site  Accuracy :  This  test’s  purpose  was  to 
demonstrate  the  accuracy  of  OMR  and 

OCR  for  about  25,000  census  form  images 
on  four  clusters  and  create  “high 
confidence”  data  files  in  the  format  used  for 
transmission  to  Census  headquarters.  The 
expected  tolerance  for  OMR  errors  was  0% 
to  1 .5%  and  the  expected  range  for  OCR 
errors  was  0%  to  2.0%. 

The  test  was  reported  successful 
although  output  data  files  were  not 
prepared  for  three  of  four  form  types. 

OMR  and  OCR  error  rates  were  captured 
for  all  four  form  types  and  ranged  from 
.35%  to  .94%  and  between  .81  %  and 

1 .45%,  respectively.  Output  data  files  for 
the  other  3  form  types  were  to  be  part  of 
the  next  SiteAT. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 
5.  Test  Events  Remain;  Completed  Events  Reported 
_ Successful  (conU _ 


Test  Name  and  Description 

Test  Results 

Load  Stability:  This  test’s  purpose  was  to 
exercise  all  clusters’  ability  to  sort  and  scan 
census  forms  over  three  8  hour  shifts. 

During  each  shift,  13  clusters’  operations 
are  simulated  while  one  cluster  is  operated 
by  test  personnel  using  paper  forms. 
Simulations  are  run  at  1 1 0  percent  of 
planned  (i.e.,  normal)  operating  capacity. 

The  test  was  reported  successful  with 
noted  exceptions.  First,  while  simulated 
sorting  of  long  forms  (41 9,424)  exceeded 
the  goal  (360,000),  simulated  sorting  of 
short  forms  failed  to  meet  the  expected 
goal  due  to  an  error  in  the  sorter 
simulator.  Simulated  scanning  of  both 
short  (1 ,205,21 8)  and  long  (1 20,787) 
forms  exceeded  the  goals  of  1 ,033,1 1 6 
and  69,888,  respectively. 

Second,  while  actual  scanning  of  short 
forms  exceeded  the  form  per  shift  goal 
(24,596)  in  two  of  the  three  shifts  (29,1 51 
and  26,884),  it  did  not  in  the  third 
(22,743).  Third,  scanning  of  long  forms 
did  not  meet  the  form  per  shift  goal  in  any 
of  the  three  shifts  (952,  681 ,  and  1 ,221 
versus  a  goal  of  1 ,658). 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 

-  According  to  Lockheed  Martin’s  test  results  report  for 
Pomona’s  SiteAT,  the  test  was  reported  successful. 

-  The  Pomona  SiteAT  results  were  consistent  with  the 
Baltimore  SiteAT  results  for  the  cluster  accuracy  and 
site  accuracy  tests  and  demonstrated  improvement  for 
the  load  stability  test. 

-  The  following  are  examples  of  tests  and  results  from 
Pomona’s  SiteAT. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 
_ Successful  (conO _ 


Test  Name  and  Description 

Test  Results 

Cluster  Accuracy:  This  test’s  purpose  was 
to  demonstrate  each  individual  cluster’s 
ability  to  scan  a  small  number  of  census 
forms  and  recognize  data  on  those  forms 
within  specified  tolerances.  The  expected 
tolerance  for  optical  mark  recognition 
(OMR)  errors  was  0%  to  1 .5%  and  the 
expected  range  for  optical  character 
recognition  (OCR)  errors  was  0%  to  2.0%. 

The  test  was  reported  successful.  For 
short  forms  the  OMR  error  rate  ranged 
from  .0%  to  .21  %  and  the  OCR  error  rate 
ranged  from  .08%  to  .28%. 

Site  Accuracy:  This  test’s  purpose  was  to 
demonstrate  the  accuracy  of  OMR  and 

OCR  for  6,000  long  census  forms  on  all 
clusters.  The  expected  tolerance  for  OMR 
errors  was  0%  to  1 .5%  and  the  expected 
range  for  OCR  errors  was  0%  to  2.0%. 

The  test  was  reported  successful.  OMR 
and  OCR  error  rates  were  .69%  and 

1.16%  respectively. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 


Test  Name  and  Description 

Test  Results 

Load  Stability:  This  test’s  purpose  was  to 
exercise  all  clusters’  ability  to  sort  and  scan 
census  forms  over  three  8  hour  shifts. 

During  each  shift,  14  clusters’  operations 
are  simulated  while  one  cluster  is  operated 
by  test  personnel  using  paper  forms. 
Simulations  are  run  at  1 10  percent  of 
planned  (i.e.,  normal)  operating  capacity. 

The  test  was  reported  successful. 
Simulated  sorting  of  long  forms  (585,000) 
met  the  goal  (585,000)  and  simulated 
sorting  of  short  forms  (2,843,400)  met  the 
goal  (2,242,500).  Simulated  scanning  of 
both  short  (1,227,240)  and  long  (84,870) 
forms  exceeded  the  goals  of  1 ,033,1 1 6 
and  69,888,  respectively. 

While  actual  scanning  of  short  forms 
exceeded  the  form  per  shift  goal  (24,598) 
in  two  of  the  three  shifts  (28,638  and 
28,785),  it  did  not  in  the  third  (24,190). 

Also,  scanning  of  long  forms  exceeded 
the  1 ,664  form  per  shift  goal  in  each  of 
the  three  shifts  (1 ,983;  1 ,922;  and  2,145). 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 

-  Lockheed  Martin’s  test  results  report  for  Phoenix’s 
SiteAT  was  being  reviewed  and  analyzed  by  the 
bureau. 

-  DCS  2000  program  officials  told  us  the  Phoenix  SiteAT 
was  successful  and  preliminary  results  demonstrated 
improvement  over  the  results  from  Pomona  SiteAT. 

-  The  following  are  examples  of  tests  and  results  from 
Phoenix’s  SiteAT  as  provided  by  bureau  officials. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 
5.  Test  Events  Remain;  Completed  Events  Reported 
_ Successful  (cont.)  _ 


Test  Name  and  Description 

Test  Results 

Cluster  Accuracy:  This  test’s  purpose  was 
to  demonstrate  each  individual  cluster’s 
ability  to  scan  a  small  number  of  census 
forms  and  recognize  data  on  those  forms 
within  specified  tolerances.  The  expected 
tolerance  for  optical  mark  recognition 
(OMR)  errors  was  0%  to  1 .5%  and  the 
expected  range  for  optical  character 
recognition  (OCR)  errors  was  0%  to  2.0%. 

For  short  forms  the  OMR  error  rate 
ranged  from  .0%  to  .1 1  %  and  the  OCR 
error  rate  ranged  from  .15%  to  .28%. 

Site  Accuracy:  This  test’s  purpose  was  to 
demonstrate  the  accuracy  of  OMR  and 

OCR  for  6,000  long  census  forms  on  all 
clusters.  The  expected  tolerance  for  OMR 
errors  was  0%  to  1 .5%  and  the  expected 
range  for  OCR  errors  was  0%  to  2.0%. 

OMR  and  OCR  error  rates  were  .61  %  and 

1 .04%  respectively. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 
5.  Test  Events  Remain;  Completed  Events  Reported 
_ Successful  (conU _ 


Test  Name  and  Description 

Test  Results 

Load  Stability:  This  test’s  purpose  was  to 
exercise  all  clusters’  ability  to  sort  and  scan 
census  forms  over  three  8  hour  shifts. 

During  each  shift,  14  clusters’  operations 
are  simulated  while  one  cluster  is  operated 
by  test  personnel  using  paper  forms. 
Simulations  are  run  at  1 10  percent  of 
planned  (i.e.,  normal)  operating  capacity. 

The  results  for  simulated  sorting  and 
simulated  scanning  are  still  being 
analyzed. 

Actual  scanning  of  short  forms  exceeded 
the  form  per  shift  goal  (24,598)  in  each  of 
the  three  shifts  (32,795;  30,772;  and 

34,41 2).  Also,  scanning  of  long  forms 
exceeded  the  1 ,664  form  per  shift  goal  in 
each  of  the  three  shifts  (2,389;  2,546;  and 
2,609). 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 

-  Operations  test  and  dry  runs  (OTDRs)  are  to  be 
conducted  by  TRW,  and  witnessed  by  the  bureau,  at 
each  of  the  three  contractor-operated  DCCs.  The 
bureau  is  to  conduct  the  OTDR  at  the  Jeffersonville 
center. 

-  TRW’s  Baltimore  test  results  report  identified  the 
following  problems. 

>  The  workload  for  performing  the  “key  from  image” 
function  was  unevenly  distributed. 

>  The  “key  from  paper”  interface  was  not  configured 
efficiently,  thus  limiting  productivity. 
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GAO  Audit  Results:  DCS  2000  Status  and  Quality 

5.  Test  Events  Remain;  Completed  Events  Reported 

^ ^ . 

>  The  “check-out”  bar  code  scanner  often  misread  the 
form  identification  number. 

-  Each  of  these  problems  is  reported  as  fixed  and  is  to  be 
retested  as  part  of  a  future  OTDR. 

-  An  additional  operational  test  was  conducted  at  the 
Baltimore  DCC  by  an  independent  research 
organization  to  assess  DCS  2000’s  accuracy.  The  test 
results  showed  DCS  2000’s  short  and  long  census  form 
OCR  accuracy  was  99.4%  and  99.3%,  respectively, 
exceeding  the  98%  accuracy  goal.  Further,  DCS 
2000’s  99.6%  OMR  accuracy  for  short  and  long  forms 
exceeded  the  99%  accuracy  goal. 
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GAO  Audit  Results:  DCS  2000  Risks/Risk  Mitigators 


•  Risks 

1 .  Important  system  development,  test,  and  deployment 
activities  remain  that  will  likely  generate  more  system 
defects,  and  the  current  schedule  will  not 
accommodate  slippage. 

2.  Concurrent  system-level  test  activities  increase 
contention  for  test  resources  and  complicate  control  of 
test  baseline  system  configurations. 

•  Risk  Mitigators 

1 .  Incremental  development  and  deployment  gets  system 
into  hands  of  users  faster. 
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GAO  Audit  Results:  DCS  2000  Risks/Risk  Mitigators 
(cont.) 

2.  Formal  risk  management  program  is  in  place  to 
address  risks  proactively,  including  steps  to  mitigate 
the  possibility  of  not  completing  remaining  system 
development  activities  on  time. 

3.  An  additional  test  event  to  exercise  all  planned  system 
capabilities  in  an  operational  environment  is  to  occur 
just  prior  to  the  March  6,  2000  deadline. 

4.  Development  contractor’s  software  process  capabilities 
are  effective,  including  processes  for  configuration 
management. 
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GAO  Audit  Results:  DCS  2000  Risks 

1 .  Activities  Remain/Delays  Cannot  Be  Accommodated 

•  The  bureau  must  be  ready  to  process  census  forms  by 
March  6,  2000,  thus  leaving  only  3  months  to  complete 
DCS  2000. 

•  The  following  significant  DCS  2000  system  development 
steps  remain: 

-  development  and  deployment  of  three  releases, 
including  fixing  known  defects; 

-  completion  of  system  acceptance  testing; 

-  completion  of  one  site  acceptance  test; 

-  completion  of  two  operational  tests;  and 

-  completion  of  one  operational  test  at  all  four  sites. _ 
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GAO  Audit  Results:  DCS  2000  Risks 

1 .  Activities  Remain/Delays  Cannot  Be  Accommodated 
_ [conU _ _ _ 


•  The  available  time  between  the  last  system  release  and 
the  DCS  2000  operational  date  is  less  than  4  weeks;  thus 
the  current  schedule  cannot  tolerate  any  delays. 
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GAO  Audit  Results:  DCS  2000  Risks 

1 .  Activities  Remain/Delays  Cannot  Be  Accommodated 
_ (cont.) _ 1 _ 


Key  Event 

1999 

2000 

Qtr  1  |  Qtr  2  |  Qtr  3  |  Qtr4 

Qtr  1  |  Qtr  2  |  Qtr  3  |  Qtr  4 

Qtr  1 

System  acceptance  testing 

Release  21 

Release  22 

o  12/17/99 
<>  01/19/00 

O  02/11/00 

11/29/99  H  12/17/99 

12/06/99  M  02/04/00 

12/20/99  HHi  02/04/00 

02/22/00  |  02/25/00 

Release  23 

Site  acceptance  test  at  Jeffersonville 

OTDR  at  Phoenix 

OTDR  at  Jeffersonville 

Four  site  operations  test 

Data  capture 

Legend 

Planned  progress  ESi 

Actual  progress 

Planned  milestone  <£>  Actual  milestone  ♦ 

Page  75 


G AO/AIMD-OO-6 1  2000  Census  Data  Capture  System 


Appendix  I 

Briefing  on  the  Decennial  Census 


GAO  Audit  Results:  DCS  2000  Risks 

2.  Multiple  Test  Activities  Occurring  Concurrently 


•  Under  normal  system-level  test  progression,  system 
acceptance  testing  occurs  first,  followed  by  site 
acceptance  testing,  and  then  site  operational  testing. 
While  some  overlap  in  the  conduct  of  these  test  events  is 
normal,  the  greater  the  level  of  concurrency  among  the 
events,  the  greater  the  risk. 

•  Introducing  concurrency  of  test  events  is  one  way  of 
saving  time  and  thus  delivering  systems  sooner.  This 
practice  has  proven  successful  on  systems  where  no 
significant  problems  are  expected  during  the  test  events 
(e.g.,  FAA  Display  Channel  Complex-Rehost). 

•  The  DCS  2000  schedule  provides  for  as  many  as  three 
system-level  test  events  to  occur  concurrently. 
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GAO  Audit  Results:  DCS  2000  Risks 

2.  Multiple  Test  Activities  Occurring  Concurrently 

•  Test  concurrency  introduces  risks  that  must  be  formally 

and  carefully  managed.  For  example: 

-  Concurrency  will  increase  contention  for  test  resources, 
particularly  key  staff. 

-  Concurrency  introduces  the  possibility  of  defects  being 
found  and  corrected  independently  during  the  different 
test  activities,  resulting  in  more  than  one  baseline  test 
configuration. 

-  Concurrency  can  require  additional  time  if  severe 
defects  are  found  that  necessitate  stopping  all  affected 
test  activities,  fixing  the  baseline,  and  then  restarting 
the  tests. 
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GAO  Audit  Results:  DCS  2000  Risk  Mitigators 

1.  Incremental  Development  and  Deployment 


•  Incremental  system  development  and  deployment  is  a 
recognized  “best  practice”  used  by  leading  public  and 
private  organizations  to  mitigate  risks  on  large,  complex 
systems. 

•  As  previously  described,  DCS  2000  is  being  built, 
deployed,  and  operationally  tested  incrementally.  By 
doing  so,  the  bureau  is  able  to  learn  sooner  rather  than 
later  about  system  problems  so  that  it  can  better  manage 
their  resolution  and  better  ensure  that  needed  system 
capabilities  will  be  ready  on  time. 
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GAO  Audit  Results:  DCS  2000  Risk  Mitigators 

2.  Formal  Risk  Management  Program  in  Place 


•  Carnegie  Mellon  University’s  Software  Engineering 
Institute  (SEI)  recommends  a  formal  risk  management 
program  that  includes  activities  to  continually  identify, 
assess,  track,  control,  and  report  risks  for  acquisitions  like 
DCS  2000. 

•  The  bureau  and  Lockheed  Martin  are  jointly  managing 
DCS  2000  risk  through  a  formal  DCS  2000  risk 
management  program  that  includes  proactive 
identification  of  risks  and  analysis  of  their  impacts  and 
probabilities,  definition  of  mitigation  plans,  and  tracking 
and  reporting  of  the  plans’  implementation  and 
effectiveness. 
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GAO  Audit  Results:  DCS  2000  Risk  Mitigators 

2.  Formal  Risk  Management  Program  in  Place  (cont.) 


•  Risk  severity  is  categorized  as  high,  medium,  or  low. 


•  As  of  December  13,  1999,  five  risks  were  categorized  as 
high,  meaning  that  they  have  a  high  probability  of 
occurrence  and  a  high  program  impact.  _ 


Risk 

Mitigation  Steps 

Status 

On-site  support 
personnel  are  not 
adequately  trained. 

Lockheed  Martin  is  developing 
training  materials,  scheduling  and 
conducting  training,  and  will  rotate 
development  staff  to  the  DCCs  to 
augment  site  personnel. 

Training  materials  have 
been  developed  and  on-site 
support  personnel  are  to 
receive  training  as  they  are 
brought  onto  the  program. 
Development  staff  have 
been  identified  to  go  to  the 
sites. 

Daily  maintenance  of 
system  administration 
software  cannot  be 
performed  at  night  if 
a  third  shift  is  added. 

Census  and  Lockheed  Martin  will 
assess  alternative  ways  of 
increasing  system  performance 
other  than  adding  a  third  shift. 

Alternatives  are  being 
analyzed.  A  recommended 
solution  is  expected  by 
January  7,  2000. 
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GAO  Audit  Results:  DCS  2000  Risk  Mitigators 

2.  Formal  Risk  Management  Program  in  Place  (cont.) 


Risk 

Mitigation  Steps 

Status 

Undefined  technical 
support  roles  and 
responsibilities  may 
affect  site  operations. 

Lockheed  Martin  is  defining  the 
roles  and  responsibilities  for  the 
site  operations  team  and 
developing  procedures  and 
guides  for  use  during  operations. 
The  procedures  and  guides  are  to 
be  used  during  the  final  2  OTDRs 
and  the  4-site  operations  test. 

Roles  and  responsibilities 
are  defined  in  the  on-site 
support  procedures  and 
guides,  which  were 
completed  on  December 

10, 1999. 

Central  technical 
support  staffing  levels 
are  currently 
insufficient  to  provide 
technical  support 
coverage  during  all 
processing  shifts  at 
the  DCCs. 

Lockheed  Martin  is  seeking  a 
contract  modification  to  increase 
central  technical  support  staffing. 

Lockheed  Martin  is 
developing  a  proposal  and 
associated  cost  estimate  for 
increased  central  technical 
support.  The  proposal  and 
cost  estimate  is  expected  to 
be  provided  to  the  Bureau 
in  January  2000. 
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GAO  Audit  Results:  DCS  2000  Risk  Mitigators 

2.  Formal  Risk  Management  Program  in  Place  (cont.) 


Risk 

Mitigation  Steps 

Status 

Tight  development 
schedules  poses  risk 
to  successful  data 
capture. 

1)  Extend  release  management 
function.  2)  Increase  focus  on 
PTRs  to  ensure  proper  priority 
and  planning  for  the  remaining 
schedule.  Review  PTR  status 
regularly  and  establish 
contingency  plans  for  all  PTRs, 
which  may  miss  committed 
schedules.  3)  Continue  to  review 
all  new  change  requests  to 
ensure  they  are  high  priority  and 
that  they  can  be  delivered  with 
manageable  risk.  4)  Make  sure 
that  all  releases  with  new 
functionality  are  tested  during  an 
OTDR.  5)  Continue  to  develop 
and  refine  the  Rapid  Response 
process,  which  requires  a  fix  or  a 
workaround  within  24  hours. 

1)  Proposal  submitted  and 
additional  release 
management  support 
authorized.  2)  Added  staff 
to  manage  PTR  progress. 
Regular  PTR  status 
meetings  held  to  review 

PTRs  with  sites  and  other 
support  areas. 

3)  Established  Engineering 
Review  Board  to  review  all 
change  requests. 

4)  Current  release  plan 
reflects  all  new  functionality 
as  part  of  Release  22  and 
will  be  tested  during  the 
Phoenix  and  Jeffersonville 
OTDRs.  Release  23  will  be 
part  of  4-site  test  and  will 
contain  PTRs  only. 

5)  Rapid  response 
process  developed. 

65 


Page  82 


GAO/AIMD-OO  61  2000  Census  Data  Capture  System 


Appendix  I 

Briefing  on  the  Decennial  Census 


GAO  Audit  Results:  DCS  2000  Risk  Mitigators 

3.  Additional  Test  of  All  System  Capabilities  Planned 


•  A  test  to  demonstrate  simultaneous  operation  of  all  four 
DCC’s,  including  DCS  2000’s  capability  to  process 
workload  equivalent  to  that  expected  during  actual  data 
capture  operation,  is  planned  for  February  22-25,  2000. 

•  The  test  will  include  operational  testing  of  all  DCS  2000 
software  releases. 

•  The  test  will  use  the  same  system  operations  and  support 
personnel  that  will  be  on  board  for  actual  data  capture 
operations. 
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GAO  Audit  Results:  DCS  2000  Risk  Mitigators 

4.  Development  Contractor  Process  Capabilities  Are 

•  SEI  has  developed  criteria  for  determining  organizations’ 
capability  to  develop  software  effectively. 

•  SEI’s  criteria  defines  five  levels  of  development 
capability,  ranging  from  level  1  (ad  hoc  and  chaotic)  to 
level  5  (optimized). 

•  Lockheed  Martin-Mission  Systems  has  been 
independently  evaluated  as  an  SEI  Level  5  development 
organization,  and  DCS  2000  was  one  of  three  systems 
included  in  the  evaluation  determination. 
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GAO  Audit  Results:  DCS  2000  Risk  Mitigators 

4.  Development  Contractor  Process  Capabilities  Are 
_ Effective  (conU _ 


•  Among  the  effective  software  development  key  processes 
that  Lockheed  Martin-Mission  Systems  has  are 
configuration  management;  project  planning,  tracking, 
and  oversight;  and  quality  assurance. 
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The  Census  Bureau  has  made  reasonable  progress  in  acquiring  and 
deploying  DCS  2000,  and  under  normal  circumstances,  where  completion 
deadlines  can  be  changed  and  some  degree  of  schedule  slippage  is 
tolerable,  delivering  promised  DCS  2000  capabilities  would  not  represent  a 
significant  risk.  However,  the  demands  of  the  decennial  census  necessitate 
that  DCS  2000  be  developed,  tested,  deployed,  and  ready  to  operate  in  just 
3  months,  with  no  allowance  for  slippage. 

Given  DCS  2000’s  current  status  and  quality,  completing  DCS  2000  by 
March  6,  2000,  poses  considerable  risk.  In  particular,  significant  system 
functionality  remains  to  be  developed,  acceptance  tested  (e.g.,  24  percent  of 
the  total  number  of  DCS  2000  high-level  functional  requirements  have  not  yet 
been  acceptance  tested),  deployed,  and  site  tested.  Further,  a  key  measure 
of  system  quality  (trends  in  unfixed  severe  defects)  is  not  yet  moving  in  a 
direction  consistent  with  that  of  a  maturing  system.  Given  that  (1 )  a  number 
of  significant  test  events  have  yet  to  occur  and  (2)  these  yet-to-occur  events 
are  likely  to  uncover  more  defects,  the  chances  of  the  defect  trend  moving  in 
a  favorable  direction  soon  are  uncertain.  Further,  given  that  the  average 
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GAO  Conclusions  (cont.) 


time  to  fix  severe  DCS  2000  defects  over  the  last  6  months  (about  10  days) 
would  consume  the  time  between  conclusion  of  the  later  test  events  and  the 
day  that  DCS  2000  must  be  operational,  the  chances  of  all  specified  system 
capabilities  being  ready  on  time  are  reduced. 

To  meet  the  challenge  of  a  very  demanding  and  fault-intolerant  schedule,  the 
bureau  has  been  following  a  plan  that  provides  for  (1)  development  and 
testing  of  DCS  2000  in  a  series  of  increments,  called  releases,  and 
(2)  deployment  and  site  testing  of  the  releases  as  they  become  ready.  This 
concurrent  development  and  testing  approach,  while  introducing  risk  due  to 
the  concurrent  activities,  allows  early  system  capability  to  get  into  the  hands 
of  the  end  users  faster,  which  could  save  time.  Further,  while  this  concurrent 
approach  does  increase  the  risk  of  contention  for  test  resources  and 
uncontrolled  changes  to  the  system  baseline  configuration  (because  of  the 
number  of  development  and  test  baselines  being  used),  these  risks  can  be 
mitigated  by  effective  system  development  capabilities,  such  as  project  risk 
management,  configuration  management,  project  planning,  and  oversight.  In 
DCS  2000’s  case,  these  activities  are  being  performed  by  the  development 
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GAO  Conclusions  (cont.) 


contractor,  which  has  been  independently  certified  as  an  SEI  level  5  (on  a  5 
level  scale)  software  developer.  Such  contractor  capability  is  an  important 
risk  mitigator,  not  only  with  respect  to  each  of  these  activities,  but  for  all 
remaining  DCS  2000  development,  testing,  and  deployment  activities. 

Nevertheless,  the  schedule  challenge  facing  the  bureau  on  DCS  2000, 
particularly  in  light  of  where  the  system  currently  stands  and  what  remains  to 
be  accomplished,  are  formidable.  Both  bureau  and  development  contractor 
officials  agree.  Accordingly,  they  have  initiated  proactive  and  appropriate 
risk  management  steps.  While  these  steps  do  not  guarantee  success,  they 
should  minimize  the  probability  of  the  risk  occurring  and  the  impact  on  the 
program  if  it  does. 
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GAO  Appendixes 


•  Appendix  I:  Scope  and  Methodology 

•  Appendix  II:  Description  of  Other  Decennial  Census 
Systems 
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GAO  Appendix  I 

Scope  and  Methodology 


•  We  determined  DCS  2000  status  and  quality  by: 

-  Reviewing  system  development  plans,  project 
management  plans,  activity  plans  (test,  deployment, 
etc.),  project  status  reports,  program  management 
briefings,  system  functional  requirements  and 
specifications,  and  test  results  reports. 

-  Comparing  original  to  revised  plans  and  current  plans 
to  status  reports. 

-  Observing  DCS  2000  OTDR  in  Baltimore. 

-  Attending  periodic  DCS  2000  project  review  meetings 
between  the  bureau  and  the  development  contractor. 
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GAO  Appendix  I 

Scope  and  Methodology  (cont.) 


-  Interviewing  bureau  DCS  2000  project  management 
and  contractor  officials. 

-  Identifying  how  the  bureau  measured  maturity  and 
quality  of  DCS  2000  throughout  its  development  life 
cycle,  and  analyzing  these  and  other  measures  for 
indicators  of  progress  and  system  health  (e.g.,  PTR 
trends  and  age  and  test  results). 

-  Reviewing  the  scope  and  results  of  DCS  2000  testing 
activities,  including  test  plans  and  results. 
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GAO  Appendix  I 

Scope  and  Methodology  (cont.) 


•  We  determined  DCS  2000  risks  and  risk  mitigators  by: 

-  Determining  bureau  and  development  contractor  key 
management  processes  (e.g.,  requirements 
management,  configuration  management,  project 
planning,  and  project  tracking  and  oversight). 

-  Analyzing  the  results  of  status  and  quality 
determinations  vis-a-vis  DCS  2000  expectations  and 
remaining  steps  to  complete. 

-  Analyzing  plans  for  completing  DCS  2000  vis-a-vis 
generally  accepted  system  engineering  principles. 

-  Interviewing  bureau  DCS  2000  project  managers  and 
contractor  officials. 
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GAO  Appendix  I 

Scope  and  Methodology  (cont.) 


•  We  briefed  bureau  and  development  contractor  officials 
on  the  results  of  our  work  and  incorporated  updated 
information  they  provided,  as  appropriate. 

•  We  performed  our  work  at  the  bureau’s  headquarters  in 
Suitland,  MD;  the  DCS  2000  program  office  in  Lanham, 
MD;  bureau  and  development  contractor  facilities  in 
Bowie,  MD;  and  the  bureau’s  DCC  in  Baltimore,  MD  from 
June  1999  through  December  1999  in  accordance  with 
generally  accepted  government  auditing  standards. 
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Briefing  on  the  Decennial  Census 


GAO  Appendix  il 

Description  of  Decennial  Census  Systems 


•  Geographic  Support  System  provides  the  basic  census 
address  list,  maps,  and  geographic  reference  files  for  all 
census  programs,  including  the  2000  decennial  census. 

•  Pre-Appointment  Management  System/Automated 
Decennial  Administrative  Management  System  is  the 
administrative  management  system  that  supports 
temporary  bureau  employee  applicant  tracking  and 
processing,  temporary  employee  selection  records, 
recruiting  reports,  personnel  and  payroll  processing,  and 
archiving  of  historical  employment  data. 
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GAO  Appendix  II 

Description  of  Decennial  Census  Systems  (cont.) 


•  Operations  Control  System  2000  supports,  manages,  and 
controls  all  field  operations  for  Census  2000. 

•  Management  Information  System  2000  is  the  official 
source  of  management  information  on  Census  2000  and 
will  provide  information  on  scheduling,  progress  to  date, 
cost  to  date  compared  to  budget,  and  performance 
anomalies. 
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GAO  Appendix  II 

Description  of  Decennial  Census  Systems  (cont.) 


•  Telephone  Questionnaire  Assistance  and  Coverage 
Edit  Follow-Up 

-  TQA  is  a  telephone  response  system  that  uses 
both  automated  interactive  voice  response 
technology  and  live  operator  responses  to  ensure 
that  the  public’s  needs  are  addressed. 

-  CEFU  flags  edit  failures  on  census  forms  and 
sends  them  to  a  telephone  follow-up  operation 
where  a  generic  follow-up  interview  will  be 
conducted. 
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GAO  Appendix  II 

Description  of  Decennial  Census  Systems  (cont.) 


•  Internet  Data  Collection/Internet  Questionnaire  Assistance 

-  IDC  provides  respondents,  on  a  limited  basis,  the  ability 
to  complete  their  short  form  questionnaire  on-line  over 
the  Internet. 

-  IQA  provides  respondents  with  answers  to  questions  via 
the  Internet. 

•  Accuracy  and  Coverage  Evaluation  is  a  follow-up  survey 
for  a  representative  sample  of  300,000  housing  units 
across  the  nation. 
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GAO  Appendix  II 

Description  of  Decennial  Census  Systems  (cont.) 


•  Headquarters  processing  is  a  group  of  systems, 
applications,  and  processes  performed  on  the  data 
collected  by  various  Census  2000  systems  during  the 
census  and  during  the  ACE  phases.  Headquarters 
processing  is  also  responsible  for  creating  final  census 
data  files. 

•  Data  Access  and  Dissemination  System  consists  of  two 
primary  subsystems,  American  FactFinder  and  Data 
Products  Production.  These  subsystems  jointly  provide 
access  to  census  results  through  the  Internet. 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


ris 


THE  SECRETARY  OF  COMMERCE 

Washington.  D.C.  20230 


J»  2  T  2000 


Mr.  Jeffrey  C.  Steinhoff 
Acting  Assistant  Comptroller  General 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Steinhoff: 

The  Department  of  Commerce  appreciates  the  opportunity  to  comment  on  the  U.S. 
General  Accounting  Office  draft  document  entitled,  2000  Census:  New  Data  Capture 
System  Progress  and  Risks.  The  U.S.  Census  Bureau’s  comments  on  this  report  are 
enclosed. 


Sincerely, 


William  M.  Daley 


Enclosure 
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See  comment  1 . 


See  comment  2. 


Enclosure 

Comments  from  the  U.S.  Department  of  Commerce 
Bureau  of  the  Census 

U.S.  General  Accounting  Office  Report  Entitled  “2000  Census:  New  Data  Capture 
System  Progress  and  Risks” 

The  U.S.  Census  Bureau  is  in  general  agreement  with  the  observations  and  conclusions 
outlined  by  the  U.S.  General  Accounting  Office  (GAO)  in  this  report.  Census  Bureau 
officials  agree  that  the  Census  2000  schedule  is  tight,  and  that  the  Data  Capture  System 
(DCS)  2000  must  meet  significant  challenges  associated  with  the  successful  completion 
of  Census  2000.  Appropriate  risk  management  procedures  have  been  initiated,  therefore, 
to  ensure  that  the  continued  implementation  of  the  DCS  2000  is  on  schedule,  and  that  the 
system  fulfills  all  of  its  operational  requirements. 

However,  the  Census  Bureau  does  have  some  comments  on  the  report.  The  GAO  states 
that  “the  demands  of  the  decennial  census  necessitate  that  DCS  2000  be  developed, 
tested,  deployed,  and  ready  to  operate  in  just  2  months.”  The  Census  Bureau  is 
concerned  that  this  sentence  implies  that  all  of  the  DCS  2000  system  must  be  developed, 
tested  and  deployed  in  2  months.  As  the  GAO  documents  in  its  report,  one  of  the 
strengths  of  the  DCS  2000  system  is  that  it  has  been  incrementally  developed  and  tested. 
In  fact,  the  Census  Bureau  is  in  the  final  stages  of  testing,  and  only  a  small  portion  of 
development  remains  to  be  completed  before  the  DCS  2000  system  will  be  fully 
operational. 

In  addition,  the  GAO  states  that  “the  time  left  between  the  conclusion  of  the  last  test 
event,  scheduled  for  February  25,  2000,  and  the  day  that  DCS  2000  must  be  operational 
is  9  days  whereas  it  has  been  taking  an  average  of  16  days  to  resolve  all  [Program 
Trouble  Reports  (PTRs)]  over  the  last  6  months. . .”  In  fact,  the  PTR  resolution  rates 
identified  in  the  report  occurred  during  periods  of  significant  software  development,  and 
not  during  periods  of  dedicated  software  maintenance.  During  the  software  development 
period,  PTR  resolution  was  not  given  the  priority  it  now  receives.  Census  Bureau 
officials  believe,  therefore,  that  the  “average  of  16  days”  cited  in  the  report  is  not  a  true 
representation  of  the  time  it  takes  to  resolve  emergency  PTRs.  Moreover,  as  noted  by 
GAO,  a  “rapid  response  process”  is  in  place  to  deal  with  emergencies.  When  this  process 
was  used  for  7  emergency  trouble  reports  that  occurred  during  the  operational  testing  dry 
run,  the  resolution  rate  was  only  2  days.  This  constitutes  a  much  more  accurate 
indication  of  the  time  it  will  take  to  resolve  any  future  problems  that  may  arise. 

Except  for  these  concerns,  the  Census  Bureau  shares  the  GAO’s  awareness  of  the 
significant  challenges  associated  with  the  unprecedented  use  of  information  technology  in 
meeting  Census  2000  data  capture  requirements.  The  GAO’s  insights  and  contributions 
in  the  development  of  this  report  are  appreciated,  and  Census  Bureau  officials  are 
confident  that  the  implementation  of  DCS  2000  is  on  schedule. 
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The  following  are  GAO’s  comments  on  the  Department  of  Commerce’s 
January  27,  2000,  letter  responding  to  a  draft  of  this  report. 


GAO  Comments 


It  was  not  our  intent  to  imply  this.  As  we  discuss  throughout  the  report, 
DCS  2000  development,  testing,  and  deployment  has  been  ongoing  on 
an  incremental  basis  and  we  clearly  identify  what  activities  have  been 
accomplished  to  date.  Further,  we  identify  what  development,  test,  and 
deployment  activities  remain  to  be  accomplished,  and  we  characterize 
these  remaining  activities  as  significant  and  formidable  in  light  of  the 
short  time  remaining  before  DCS  2000  must  be  operational. 
Nevertheless,  we  have  modified  the  sentence  that  the  bureau  cited  to 
eliminate  the  possibility  of  misinterpretation. 


2.  We  agree  that  the  16-day  average  time  for  resolving  all  PTRs  is  not  a 
good  indicator  of  the  time  necessary  to  resolve  emergency  PTRs 
because  all  PTRs  include  five  different  categories  of  PTRs  and  each 
category  receives  a  different  priority,  emergency  being  the  most  severe 
category  and  the  highest  priority  for  resolution.  For  this  reason,  we  do 
not  state  anywhere  in  our  report  that  the  16-day  average  time  needed 
over  the  last  6  months  (which  covered  June  through  November  1999)  to 
resolve  all  PTRs  is  the  operative  indicator  of  the  time  to  resolve  the 
most  severe  category  of  PTRs  (emergency) .  Rather,  the  operative 
indicator  that  we  cite  in  the  report  for  the  time  necessary  to  resolve 
severe  PTRs  is  10  days,  which  is  the  average  time  that  the  bureau’s  own 
data  show  it  has  taken  over  the  last  6  months  to  resolve  both 
emergency  and  critical  PTRs.  We  focused  on  these  two  most  severe 
categories  of  PTRs  in  our  analysis  because  the  bureau’s  own  definition 
of  emergency  and  critical  PTRs  (emergency  PTRs  are  problems  that 
prevent  system  operation  and  critical  PTRs  are  problems  that 
significantly  degrade  system  operation)  mean  that  both  of  these 
categories  of  PTRs  will  have  to  be  resolved  before  March  6,  2000,  if 
DCS  2000  is  to  operate  as  intended.  Accordingly,  we  state  in  the  report 
that  “it  has  been  taking  an  average  of  16  days  to  resolve  all  PTRs  over 
the  last  6  months  and  10  days  for  severe  PTRs.” 


(511178) 


Page  101 


GAO/AIMD -00-61  2000  Census  Data  Capture  System 


Ordering  Information 


The  first  copy  of  each  GAO  report  and  testimony  is  free. 

Additional  copies  are  $2  each.  Orders  should  be  sent  to  the 
following  address,  accompanied  by  a  check  or  money  order  made 
out  to  the  Superintendent  of  Documents,  when  necessary,  VISA  and 
MasterCard  credit  cards  are  accepted,  also. 

Orders  for  100  or  more  copies  to  be  mailed  to  a  single  address  are 
discounted  25  percent. 

Orders  by  mail: 

U.S.  General  Accounting  Office 
P.O.  Box  37050 
Washington,  DC  20013 

or  visit: 

Room  1100 

700  4th  St.  NW  (corner  of  4th  and  G  Sts.  NW) 

U.S.  General  Accounting  Office 
Washington,  DC 

Orders  may  also  be  placed  by  calling  (202)  512-6000 

or  by  using  fax  number  (202)  512-6061,  or  TDD  (202)  512-2537. 

Each  day,  GAO  issues  a  list  of  newly  available  reports  and 
testimony.  To  receive  facsimile  copies  of  the  daily  list  or  any  list 
from  the  past  30  days,  please  call  (202)  512-6000  using  a  touchtone 
phone.  A  recorded  menu  will  provide  information  on  how  to  obtain 
these  lists. 

For  information  on  how  to  access  GAO  reports  on  the  INTERNET, 
send  an  e-mail  message  with  “info”  in  the  body  to: 

info@www.gao.gov 

or  visit  GAO’s  World  Wide  Web  Home  Page  at: 
http://www.gao.gov 
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